Bad script performance

Hi,

my task was to write a script that iterates every project in the database, and every item in the project. Then it should check the access of a given group to every item. The complete database contains about 10k items.

The script works, but it works very slowly and charges the full capacity of physical memory. Another problem is that the pysical memory is not deleted after runtime.

Can someone help me to increase my scripts performance?
(script attached)

Best Regards

Manuel
ManuelFelger - Fri Feb 27 06:42:09 EST 2009

Re: Bad script performance
ManuelFelger - Fri Feb 27 06:43:40 EST 2009

(script)
Attachments

attachment_14222482_iterate_Database.dxl

Re: Bad script performance
tommy3824 - Fri Feb 27 08:22:56 EST 2009

ManuelFelger - Fri Feb 27 06:43:40 EST 2009
(script)

I would start put putting in there somewhere to close the module after you open it. That is where all the memory is going. You don't notice it though because you open them without them being visible. I would bet your script would run a lot faster if you close the module immediately after it gets the information it needs. A helpful function to is the 'isVisible (Module m)' so you don't close any modules you physically opened.

Re: Bad script performance
ManuelFelger - Fri Feb 27 08:40:25 EST 2009

tommy3824 - Fri Feb 27 08:22:56 EST 2009
I would start put putting in there somewhere to close the module after you open it. That is where all the memory is going. You don't notice it though because you open them without them being visible. I would bet your script would run a lot faster if you close the module immediately after it gets the information it needs. A helpful function to is the 'isVisible (Module m)' so you don't close any modules you physically opened.

I dont know if i understood what you mean, where do i open modules? is it the "ar = get(m2, gName, errMsg)" function that opens the modules?

Re: Bad script performance
ePiallat - Fri Feb 27 08:58:48 EST 2009

ManuelFelger - Fri Feb 27 08:40:25 EST 2009
I dont know if i understood what you mean, where do i open modules? is it the "ar = get(m2, gName, errMsg)" function that opens the modules?

Hi,

The "load" perm opens the module.

But in fact, you never needed to open it to get an access record. You may directly get them from the "Item" variable. And as you looking for AccessRec for every item whatever the type, you can simplify (and accelerate) your code with:
 

string checkAccess(string gName, string targetName)
{
    string errMsg
        AccessRec ar
        
        Item i = item (targetName)
        ar = get(i, gName, errMsg)

 


You may also have issues if there is an access record denying access to your group. In this case, your "ar" won't be null, but (read ar) and (write ar) will be false, so your function won't return anything and throw a DXL error.

You'd better add some else return "None" at th end of your function.
Finally, if you're looking for actual access rights, including inherited ones, consider using the "hasPermission" perm instead of "AccessRec get". It is documented for "Signature" only, but works the same for any Item, Module, Folder, Group, ModName_, usw.

 

Re: Bad script performance
ManuelFelger - Fri Feb 27 10:21:03 EST 2009

ePiallat - Fri Feb 27 08:58:48 EST 2009

Hi,

The "load" perm opens the module.

But in fact, you never needed to open it to get an access record. You may directly get them from the "Item" variable. And as you looking for AccessRec for every item whatever the type, you can simplify (and accelerate) your code with:
 

string checkAccess(string gName, string targetName)
{
    string errMsg
        AccessRec ar
        
        Item i = item (targetName)
        ar = get(i, gName, errMsg)

 


You may also have issues if there is an access record denying access to your group. In this case, your "ar" won't be null, but (read ar) and (write ar) will be false, so your function won't return anything and throw a DXL error.

You'd better add some else return "None" at th end of your function.
Finally, if you're looking for actual access rights, including inherited ones, consider using the "hasPermission" perm instead of "AccessRec get". It is documented for "Signature" only, but works the same for any Item, Module, Folder, Group, ModName_, usw.

 

Hi,

thank you for leting me know i've opened the modules, and thank you for the tip to get the access information directly from the item. Script run less than 10 sec now.

But i didnt understand what the difference between the "hasPermission" perm and "AccessRec get" is.

Regards
Manuel

Re: Bad script performance
ManuelFelger - Fri Feb 27 10:27:37 EST 2009

ManuelFelger - Fri Feb 27 10:21:03 EST 2009
Hi,

thank you for leting me know i've opened the modules, and thank you for the tip to get the access information directly from the item. Script run less than 10 sec now.

But i didnt understand what the difference between the "hasPermission" perm and "AccessRec get" is.

Regards
Manuel

(updated script)
Attachments

attachment_14222572_1_1.dxl

Re: Bad script performance
ePiallat - Fri Feb 27 11:05:26 EST 2009

ManuelFelger - Fri Feb 27 10:21:03 EST 2009
Hi,

thank you for leting me know i've opened the modules, and thank you for the tip to get the access information directly from the item. Script run less than 10 sec now.

But i didnt understand what the difference between the "hasPermission" perm and "AccessRec get" is.

Regards
Manuel

My bad... "hasPermission" is only usefull (and useable) to survey User's access rights.

As you are listing accesses of a group, you did right and don't have to modify anything.

But for further reference, "AccessRec" are the rules listed in the "Access" tab. But if you want permission granted to an user, you need a combination of his access record, access records of any group on which he has membership, and the default (Everyone else) access record.

This combination is simply computed by the "hasPermission" perm. (And permits DOORS printing your personnal access rights at the bottom of each Access record list in graphic interface, even if you're not listed in access records).

– Éric

Re: Bad script performance
ManuelFelger - Fri Feb 27 11:31:55 EST 2009

ePiallat - Fri Feb 27 11:05:26 EST 2009
My bad... "hasPermission" is only usefull (and useable) to survey User's access rights.

As you are listing accesses of a group, you did right and don't have to modify anything.

But for further reference, "AccessRec" are the rules listed in the "Access" tab. But if you want permission granted to an user, you need a combination of his access record, access records of any group on which he has membership, and the default (Everyone else) access record.

This combination is simply computed by the "hasPermission" perm. (And permits DOORS printing your personnal access rights at the bottom of each Access record list in graphic interface, even if you're not listed in access records).

– Éric

The next tast of the script should be to survey user's access rights, i thought about something like this:

User u = find(userName)
Group g

//GROUP FINDING LOOP
for g in groupList do
{
if (member (g, u))
{
(here would be my acutal script)
}
}

is there any more effective way?
How do i use the "hasPermission" function in this case?

Manuel

Re: Bad script performance
llandale - Sun Mar 01 15:06:52 EST 2009

ManuelFelger - Fri Feb 27 11:31:55 EST 2009
The next tast of the script should be to survey user's access rights, i thought about something like this:

User u = find(userName)
Group g

//GROUP FINDING LOOP
for g in groupList do
{
if (member (g, u))
{
(here would be my acutal script)
}
}

is there any more effective way?
How do i use the "hasPermission" function in this case?

Manuel

Yup, no user-in-group loop. Odd.

So yes, get a list of all groups to which the user belongs. For each item (module), loop through the access records looking for either the specific user, the user's groups, or the default everyone else record (the name of that one is null). Accumulate the rights provided by the Groups of that user. If there is a specific record for that user then those are the rights, stop looking and ignore whatever groups rights you've accumulated. If there is no user nor group access records, use the default.

Its tricky; as is how to accumulate Permissions for a particular access record.

>Louie

Re: Bad script performance
ePiallat - Mon Mar 02 03:20:18 EST 2009

llandale - Sun Mar 01 15:06:52 EST 2009
Yup, no user-in-group loop. Odd.

So yes, get a list of all groups to which the user belongs. For each item (module), loop through the access records looking for either the specific user, the user's groups, or the default everyone else record (the name of that one is null). Accumulate the rights provided by the Groups of that user. If there is a specific record for that user then those are the rights, stop looking and ignore whatever groups rights you've accumulated. If there is no user nor group access records, use the default.

Its tricky; as is how to accumulate Permissions for a particular access record.

>Louie

Please try this - it illustrates the use of "hasPermission":

string getPermissions (string uName, Item i) {
if (null i) return "Null Item"
if (!existsUser uName) return "User not found"

if (!hasPermission (uName, i, read)) return "None"

if (hasPermission (uName, i, control)) return "RMCDA"
if (hasPermission (uName, i, create|delete)) return "RMCD"
if (hasPermission (uName, i, delete)) return "RMD"
if (hasPermission (uName, i, create|modify)) return "RMC"
if (hasPermission (uName, i, create)) return "RC"
if (hasPermission (uName, i, modify)) return "RM"

return "R"
}

print getPermissions ("userName", item "fullName/of/module")

Re: Bad script performance
ManuelFelger - Mon Mar 02 05:08:03 EST 2009

ePiallat - Mon Mar 02 03:20:18 EST 2009
Please try this - it illustrates the use of "hasPermission":

string getPermissions (string uName, Item i) {
if (null i) return "Null Item"
if (!existsUser uName) return "User not found"

if (!hasPermission (uName, i, read)) return "None"

if (hasPermission (uName, i, control)) return "RMCDA"
if (hasPermission (uName, i, create|delete)) return "RMCD"
if (hasPermission (uName, i, delete)) return "RMD"
if (hasPermission (uName, i, create|modify)) return "RMC"
if (hasPermission (uName, i, create)) return "RC"
if (hasPermission (uName, i, modify)) return "RM"

return "R"
}

print getPermissions ("userName", item "fullName/of/module")

Thank you Eric,

this helps to get Access Rights more exactly. But now i've already a new question :-)
In my Telelogic DXL manual in chapter "Reading from a File" is only explained how to read a file line by line.
Is there a way to read *.csv files cell by cell?

Manuel

Re: Bad script performance
Tony_Goodman - Mon Mar 02 08:53:32 EST 2009

ManuelFelger - Mon Mar 02 05:08:03 EST 2009
Thank you Eric,

this helps to get Access Rights more exactly. But now i've already a new question :-)
In my Telelogic DXL manual in chapter "Reading from a File" is only explained how to read a file line by line.
Is there a way to read *.csv files cell by cell?

Manuel

The attached might help.
It is a CSV parser utility that allows you to parse any CSV file and handle each column entry individually.

It is intended to be simple to use and is not efficient for very large files.

Tony
Attachments

attachment_14223232_smartCsvParser.inc

Re: Bad script performance
tommy3824 - Mon Mar 02 09:25:47 EST 2009

ManuelFelger - Mon Mar 02 05:08:03 EST 2009
Thank you Eric,

this helps to get Access Rights more exactly. But now i've already a new question :-)
In my Telelogic DXL manual in chapter "Reading from a File" is only explained how to read a file line by line.
Is there a way to read *.csv files cell by cell?

Manuel

use regualar expression...

string str = "one,line,of,text"
if(matches("(,+),(,+),(,+),(^,+)$",str))
{
print strmatch 0"\n" // prints the whole string going in
print strmatch 1"\n" // prints one
print strmatch 2"" // prints line
//... so on
}

I'm pretty sure you can only go up to match 9 too. This expression also will only work if the format matches this exactly (otherwise the if fails). I usually to a lot of format checking before reading this into a function to prevent any runtime errors further down (such as null values,whitespace, and file ending with a blank line). Hope this helps.

Re: Bad script performance
tommy3824 - Mon Mar 02 09:29:18 EST 2009

Tony_Goodman - Mon Mar 02 08:53:32 EST 2009
The attached might help.
It is a CSV parser utility that allows you to parse any CSV file and handle each column entry individually.

It is intended to be simple to use and is not efficient for very large files.

Tony

oops, guess I had my window opened too long. Didn't see your reply.

PS the links are suppose to be square brackets but looks like the forum had its way with them

Re: Bad script performance
ManuelFelger - Mon Mar 09 06:25:04 EDT 2009

tommy3824 - Mon Mar 02 09:29:18 EST 2009
oops, guess I had my window opened too long. Didn't see your reply.

PS the links are suppose to be square brackets but looks like the forum had its way with them

I just tried to read a *.csv testfile to see in which format the lines will be printed, but nothing happens..

If i read a *.txt file the same way it works, what do i have to change to read *.csv ?
 

string fileName = "d:\\list.csv"
 
Stream input = read fileName
 
string groupName
 
for (input >> groupName; ! end input; input >> groupName)
{
    print groupName"\n"
}
 
close input

Re: Bad script performance
ManuelFelger - Mon Mar 09 09:19:07 EDT 2009

ManuelFelger - Mon Mar 09 06:25:04 EDT 2009

I just tried to read a *.csv testfile to see in which format the lines will be printed, but nothing happens..

If i read a *.txt file the same way it works, what do i have to change to read *.csv ?
 

string fileName = "d:\\list.csv"
 
Stream input = read fileName
 
string groupName
 
for (input >> groupName; ! end input; input >> groupName)
{
    print groupName"\n"
}
 
close input

Now reading *.csv files works, but i dont know how to use the matches() command.

How to split a string at every semicolon?

Something like this could be the string coming out the *.csv file:

"L6/General;L6/Analysis/;L6/Requirements"

how to split that string in:

"L6/General"
"L6/Analysis"
"L6/Requirements"

Can someone help me?

Manuel

Re: Bad script performance
ManuelFelger - Mon Mar 09 10:03:35 EDT 2009

ManuelFelger - Mon Mar 09 09:19:07 EDT 2009
Now reading *.csv files works, but i dont know how to use the matches() command.

How to split a string at every semicolon?

Something like this could be the string coming out the *.csv file:

"L6/General;L6/Analysis/;L6/Requirements"

how to split that string in:

"L6/General"
"L6/Analysis"
"L6/Requirements"

Can someone help me?

Manuel

I've found out myself..problem solved

Re: Bad script performance
strathglass - Thu May 14 08:45:03 EDT 2009

Tony_Goodman - Mon Mar 02 08:53:32 EST 2009
The attached might help.
It is a CSV parser utility that allows you to parse any CSV file and handle each column entry individually.

It is intended to be simple to use and is not efficient for very large files.

Tony

Hey Tony - that CSV parser works great, thanks for posting it!
(Just found it in a search and it does exactly what I need!)

Regards,
Strathglass.